Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dse netty 4.1.34.final #18

Open
wants to merge 420 commits into
base: dse-netty-4.1.25.Final
Choose a base branch
from

Conversation

sbtourist
Copy link

No description provided.

normanmaurer and others added 30 commits September 14, 2018 19:01
… version. (netty#8289)

Motivation:

6ed7c6c added a test which blindly assumed we can use a KeyManagerFactory all the time. This is only true if have OpenSSL 1.0.2 or later, which may not be the case.

Modifications:

Only use KeyManagerFactory in test if the OpenSSL version does support it.

Result:

More robust tests.
…ows an UnsupportedOperationException. (netty#8290)

Motivation:

6ed7c6c added support for ExtendedOpenSslSession but we did not override getStatusResponses(). This lead to test failures on java9.

Modifications:

Implement ExtendedOpenSslSession.getStatusResponses() so it just returns an empty list.

Result:

Test pass again on Java9.
…tty#8286)

Motivation:

Get charset from Content-Type header even it contains multiple parameters.

Modification:

Extract charset value from the charset parameter if it is not last.

Result:

Fixes netty#8273
Motivation:

Conscrypt 1.3.0 was just released and adds support for TLSv1.3

Modifications:

Update to 1.3.0

Result:

Use latest conscrypt during build / test.
Motivation:

If you encode a SOCKS5 message like new DefaultSocks5CommandResponse(FAILURE, DOMAIN, "", 0)
you correctly get a result of 05010003000000.
But if the bndAddr is null, for example like new DefaultSocks5CommandResponse(FAILURE, DOMAIN)
the encoded result is 0501000301000000 which means the domain name has a length of one and consists of a 0-byte.

Modification:

With this commit it is also correctly encoded as a string of 0 length.

Result:

Correctly encode empty SOCKS5 address
…enceCountedOpenSslEngine (netty#8297)

Motivation:

Java9 added getStatusResponses() to ExtendedSSLSession which we should correctly support when possible.

Modifications:

Implement the method correctly.

Result:

More complete and correct implementation.
Motivation:

I noticed that we had some errors showing up in a test (which did not fail it tho) because we tried to full-fill the promise multiples times.

Modifications:

Use trySuccess(...) as we may produce multiple exceptions.

Result:

Less errors during test-run.
Motivation:

In theory our estimation of the needed buffer could be off and so we need to ensure we grow it if there is no space left.

Modifications:

Ensure we grow the buffer if there is no space left in there but we still have data to deflate.

Result:

Correctly deflate data in all cases.
Motivation:

We should use final Java11 release during builds.

Modifications:

Update to final Java11 release

Result:

Use latest release.
Motivation:

We need to release the ReferenceCountedSslContext to eliminate resource leaks.

Reported in https://garage.netty.io/teamcity/viewLog.html?buildId=33353&buildTypeId=netty_build_oraclejdk8&tab=buildLog#_focus=157264.

Modifications:

Call release on the SslContext instances.

Result:

No more leaks in tests.
Motivation:

First EA releases of Java12 are out we should be able to compile with these and run tests.

Modifications:

Add maven profile for java12.

Result:

Be able to use Java12
netty#8318) (netty#8319)

Motivation

Applications should not depend on internal packages with Java 9 and later. This cause a warning now, but will break in future versions of Java.

Modification

This change adds methods to UnixResolverDnsServerAddressStreamProvider (following after netty#6844) that parse /etc/resolv.conf for domain and search entries. Then DnsNameResolver does not need to rely on sun.net.dns.ResolverConfiguration to do this.

Result

Fixes netty#8318. Furthermore, at least in my testing with Java 11, this also makes multiple search entries work properly (previously I was only getting the first entry).
…in buffer. (netty#8325)

Motivation:

We need to ensure the Cumulator always releases the input buffer if it can not take over the ownership of it as otherwise it may leak.

Modifications:

- Correctly ensure the buffer is always released.
- Add unit tests.

Result:

Ensure buffer is always released.
Motivation:

We should add some command to be able to run all tests with leak detection enabled. This will then be used on the CI during PR builds.

Modifications:

Add new docker-compose config to run with leak-detection enabled.

Result:

Easy way to enable leak detection while running tests via docker.
Motivation:

4d14586 did fix some leaks in SniClientTest but missed the ones in SniClientJava8TestUtil.

Modifications:

Correctly release SslContext.

Result:

No more leaks in SNI tests.
Motivation:

The first EA builds for Java12 are released so we should allow to run with these in our docker-compose setup.

Modifications:

Add docker-compose configs for Java12.

Result:

Be able to run easily with Java12 as well.
…etty#8314)

* Add cache for CNAME mappings resolved during lookup of DNS entries.

Motivation:

If the CNAMEd hostname is backed by load balancing component, typically the final A or AAAA DNS records have small TTL. However, the CNAME record itself is setup with longer TTL.

For example:
* x.netty.io could be CNAMEd to y.netty.io with TTL of 5 min
* A / AAAA records for y.netty.io has a TTL of 0.5 min

In current Netty implementation, original hostname is saved in resolved cached with the TTL of final A / AAAA records. When that cache entry expires, Netty recursive resolver sends at least two queries — 1st one to be resolved as CNAME record and the 2nd one to resolve the hostname in CNAME record.
If CNAME record was cached, only the 2nd query would be needed most of the time. 1st query would be needed less frequently.

Modifications:

Add a new CnameCache that will be used to cache CNAMEs and so may reduce queries.

Result:

Less queries needed when CNAME is used.
…netty#8316)

* Use AuthoritativeDnsServerCache for creating the new redirect stream.

Motivation:

At the moment if a user wants to provide custom sorting of the nameservers used for redirects it needs to be implemented in two places. This is more complicated as it needs to be.

Modifications:

- Just delegate to the AuthoritativeDnsServerCache always as we fill it before we call newRedirectDnsServerStream anyway.

Result:

Easier way for the user to implement custom sorting.
…d with the OpenSSL provider. (netty#8307)

Motivation:

When a X509TrustManager is used while configure the SslContext the JDK automatically does some extra checks during validation of provided certs by the remote peer. We should do the same when our native implementation is used.

Modification:

- Automatically wrap a X509TrustManager and so do the same validations as the JDK does.
- Add unit tests.

Result:

More consistent behaviour. Fixes netty#6664.
…t. (netty#8333)

Motivation:

a208f6d added a testcase which uses hostname validation which may not be supported by OpenSSL depending on the version that is used. We should check first before we try to use it.

Modifications:

Add assumeTrue(...) check to ensure hostname validation is supported before trying to run the test.

Result:

No more test-failures on OpenSSL versions < 1.0.2.
…ng server-side and support more methods of ExtendedSSLSession. (netty#8283)

Motivation:

Before when on server-side we just called the X509KeyManager methods when handshake() was called the first time which is not quite correct as we may not have received the full SSL hello / handshake and so could not extra for example the SNI hostname that was requested.
OpenSSL exposes the SSL_CTX_set_cert_cb function which allows to set a callback which is executed at the correct moment, so we should use it. This also allows us to support more methods of ExtendedSSLSession easily.

Modifications:

- Make use of new methods exposed by netty-tcnative since netty/netty-tcnative#388 to ensure we select the key material at the correct time.
- Implement more methods of ExtendedOpenSslSession
- Add unit tests to ensure we are able to retrieve various things on server-side in the X509KeyManager and so verify it is called at the correct time.
- Simplify code by using new netty-tcnative methods.

Result:

More correct implementation for server-side usage and more complete implemented of ExtendedSSLSession.
Motivation

The EpollChannelConfig (same for KQueues) and its subclasses repeatetly declare their own channel field which leads to a 3x repetition for each config instance. Given the fields are protected or package-private it's exposing the code code to "field hiding" bugs.

Modifications

Use the the existing protected channel field from the DefaultChannelConfig class and simply cast it when needed.

Result

Fixes netty#8331
…ailure (netty#8332)

Motivation:

When writing an HTTP/2 HEADERS with END_STREAM=1, the application expects
the stream to be closed afterward. However, the write can fail locally
due to HPACK encoder and similar. When that happens we need to make sure
to issue a RST_STREAM otherwise the stream can be closed locally but
orphaned remotely. The RST_STREAM is typically handled by
Http2ConnectionHandler.onStreamError, which will only send a RST_STREAM
if that stream still exists locally.

There are two possible flows for trailers, one handled immediately and
one going through the flow controller. Previously they behaved
differently, with the immedate code calling the error handler after
closing the stream. The immediate code also used a listener for calling
closeStreamLocal while the flow controlled code did so immediately after
the write.

The two code paths also differed in their VoidChannelPromise handling,
but both were broken. The immediate code path called unvoid() only if
END_STREAM=1, however it could always potentially add a listener via
notifyLifecycleManagerOnError(). And the flow controlled code path
unvoided incorrectly, changing the promise completion behavior. It also
passed the wrong promise to closeStreamLocal() in FlowControlledBase.

Modifications:

Move closeStreamLocal handling after calls to onError. This is the
primary change.

Now call closeStreamLocal immediately instead of when the future
completes. This is the more likely correct behavior as it matches that
of DATA frames.

Fix all the VoidChannelPromise handling.

Result:

Http2ConnectionHandler.onStreamError sees the same state as the remote
and issues a RST_STREAM, properly cleaning up the stream.
Motivation:

Add an option (through a SelectStrategy return code) to have the Netty event loop thread to do busy-wait on the epoll.

The reason for this change is to avoid the context switch cost that comes when the event loop thread is blocked on the epoll_wait() call.

On average, the context switch has a penalty of ~13usec.

This benefits both:

The latency when reading from a socket
Scheduling tasks to be executed on the event loop thread.
The tradeoff, when enabling this feature, is that the event loop thread will be using 100% cpu, even when inactive.

Modification:

Added SelectStrategy option to return BUSY_WAIT
Epoll loop will do a epoll_wait() with no timeout
Use pause instruction to hint to processor that we're in a busy loop
Result:

When enabled, minimizes impact of context switch in the critical path
)

Motivation:

No need in varargs, the method always accepts array.

Modification:

... replaced with []
Motivation:

Unless the 'io.netty.noKeySetOptimization' system property is set,
registering a SelectableChannel instance to a NioEventLoop results
in a ClassCastException:

    io.netty.channel.nio.SelectedSelectionKeySetSelector cannot be cast
        to java.nio.channels.spi.AbstractSelector

Modifications:

Instead of 'selector', pass 'unwrappedSelector' to SelectableChannel.

Result:

It is possible to register a SelectableChannel instance without
setting the 'io.netty.noKeySetOptimization' system property.
…ty#8347)

Motivation:

Avoid creating any StringBuilder instance if
ByteBufInputStream::readLine isn't used

Modifications:

The StringBuilder instance is lazy allocated on demand and
are added new test case branches to address the increased
complexity of ByteBufInputStream::readLine

Result:

Reduced GC activity if ByteBufInputStream::readLine isn't used
Motivation:

Seems like IntegerHolder counterHashCode field is the very old legacy field that is no longer used. Should be marked as deprecated and removed in the future versions.

Modification:

IntegerHolder class, InternalThreadLocalMap.counterHashCode() and InternalThreadLocalMap.setCounterHashCode(IntegerHolder counterHashCode) are now deprecated.
normanmaurer and others added 21 commits March 1, 2019 06:47
…etty#8900)

Motivation:

We should run a CI job using J9 to ensure netty also works when using different JVMs.

Modifications:

- Adjust PooledByteBufAllocatorTest to be able to complete faster when using a JVM which takes longer when joining Threads (this seems to be the case with J9).
- Skip UDT tests on J9 as UDT is not supported there.

Result:

Be able to run CI against J9.
Motivation:

To ensure Netty works on different JVMs we should also run tests on the CI with these.

Modifications:

Add docker-compose config to run build with OpenJ9 JVM

Result:

Ensure Netty works with different JVMs
…engine is destroyed (netty#8905)

Motivation:

We must only remove ReferenceCountedOpenSslEngine from OpenSslEngineMap when engine is destroyed as the verifier / certificate callback may be called multiple times when the remote peer did initiate a renegotiation.
If we fail to do so we will cause an NPE like this:

```
13:16:36.750 [testsuite-oio-worker-5-18] DEBUG i.n.h.s.ReferenceCountedOpenSslServerContext - Failed to set the server-side key material
java.lang.NullPointerException: null
	at io.netty.handler.ssl.OpenSslKeyMaterialManager.setKeyMaterialServerSide(OpenSslKeyMaterialManager.java:69)
	at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext$OpenSslServerCertificateCallback.handle(ReferenceCountedOpenSslServerContext.java:212)
	at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:575)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1124)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1236)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1279)
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:217)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1330)
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
	at io.netty.channel.oio.AbstractOioByteChannel.doRead(AbstractOioByteChannel.java:170)
	at io.netty.channel.oio.AbstractOioChannel$1.run(AbstractOioChannel.java:40)
	at io.netty.channel.ThreadPerChannelEventLoop.run(ThreadPerChannelEventLoop.java:69)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
```

While the exception is kind of harmless (as we will reject the renegotiation at the end anyway) it produces some noise in the logs.

Modifications:

Don't remove engine from map after handshake is complete but wait for it to be removed until the engine is destroyed.

Result:

No more NPE and less noise in the logs.
…netty#8904)


Motivation:

Netty is very widely used which can lead to a lot of pain when we break API / ABI. We should make use japicmp-maven-plugin during the build to verify we do not introduce breakage by mistake.

Modifications:

- Add japicmp-maven-plugin to the build process
- Fix a method signature change in HttpProxyHandler that was flagged as a possible problem.

Result:

Ensure no API/ABI breakage accour between releases.
…y name. (netty#8913)

Motivation:

2bb9f64 introduced a change which made it possible to use different shaded versions of netty-tcnative on the classpath. This only partly worked as we did not correctly handled the case when os / arch is part of the library name (which is the case when netty-tcnative-boringssl-static is used with the uber jar).

Modifications:

- If patching the ID failed we retry again with the os / arch stripped
- Add unit tests to verify that patching ID now works with and without os / arch as suffix.

Result:

Using multiple shaded version of netty-tcnative-boringssl-static on MacOS works.
…tty#8859)

Motivation:

SSLEngine API has a notion of tasks that may be expensive and offload these to another thread. We did not support this when using our native implementation but can now for various operations during the handshake.

Modifications:

- Support offloading tasks during the handshake when using our native SSLEngine implementation
- Correctly handle the case when NEED_TASK is returned and nothing was consumed / produced yet

Result:

Be able to offload long running tasks from the EventLoop when using SslHandler with our native SSLEngine.
…y#8914)

Motivation:

Simple rules:

* close the connection when sending any error
* specify "Connection: close" header when closing the connection
* successful responses should keep the connection intact when otherwise is not requested by the client

Modifications:

* "send response and cleanup the connection" logic moved to a helper
* for all successful responses set "Content-Lenght" header
* do not specify "Connection: Keep-Alive" header as far it's a default for HTTP/1.1
* set "Connection: close" header when necessary

Result:

Keep-Alive connections management is inlined with RFCs.
…tions. (netty#8919)

Motivation:

In the past we found a lot of SSL related bugs because of the interopt tests we have in place between different SSLEngine implementations. We should have as many of these interopt tests as possible for this reason.

Modifications:

- Add interopt tests between Conscrypt and OpenSSL SSLEngine implementations

Result:

More tests for SSL.
netty#8921)

* Do not use GetPrimitiveArrayCritical(...) due multiple not-fixed bugs related to GCLocker

Motivation:

GetPrimitiveArrayCritical(...) may cause multiple not-fixed bugs related to the GCLocker while there is little gain for our use-case. We should just use GetByteArrayRegion(...) and copy into a small on-stack buffer.

See also:

- https://shipilev.net/jvm/anatomy-quarks/9-jni-critical-gclocker/#_g1
- https://bugs.openjdk.java.net/browse/JDK-8048556
- https://bugs.openjdk.java.net/browse/JDK-8057573
- https://bugs.openjdk.java.net/browse/JDK-8057586

Special thanks to @jayv @shipilev @apangin for the pointers.

Modifications:

Replace GetPrimitiveArrayCritical(...) with GetByteArrayRegion(...)

Result:

Less risks hitting GCLocker related bugs.
netty#8922)

Motivation:

When HttpContentDecoder (and so HttpContentDecompressor) does not produce any message we need to make sure it calls ctx.read() if auto read is false to not stale.

Modifications:

- Keep track if we need to call ctx.read() or not
- Add unit test

Result:

Fixes netty#8915.
netty#8918)


Motivation:

The SSLSession that is returned by SSLEngine.getHandshakeSession() must be able to provide the local certificates when the TrustManager is invoked on the server-side.

Modifications:

- Correctly return the local certificates
- Add unit test

Result:

Be able to obtain local certificates from handshake SSLSession during verification on the server side.
Motivation:

While looking at hpack header-processing hotspots I noticed some low
level too-big-to-inline methods which can be shrunk.

Modifications:

Reduce bytecode size and/or runtime operations used for the following
methods:

PlatformDependent0.equals(byte[], ...)
PlatformDependent0.equalsConstantTime(byte[], ...)
PlatformDependent0.hashCodeAscii(byte[],int,int)
PlatformDependent.hashCodeAscii(CharSequence)

Result:

Existing benchmarks show decent improvement

Before

Benchmark                     (size)   Mode  Cnt         Score         Error  Units
HpackUtilBenchmark.newEquals   SMALL  thrpt    5  17200229.374 ± 1701239.198  ops/s
HpackUtilBenchmark.newEquals  MEDIUM  thrpt    5   3386061.629 ±   72264.685  ops/s
HpackUtilBenchmark.newEquals   LARGE  thrpt    5    507579.209 ±   65883.951  ops/s

After

Benchmark                     (size)   Mode  Cnt         Score         Error  Units
HpackUtilBenchmark.newEquals   SMALL  thrpt    5  29221527.058 ± 4805825.836  ops/s
HpackUtilBenchmark.newEquals  MEDIUM  thrpt    5   6556251.645 ±  466115.199  ops/s
HpackUtilBenchmark.newEquals   LARGE  thrpt    5    879828.889 ±  148136.641  ops/s

Before

Benchmark                          (size)  Mode  Cnt     Score     Error  Units
PlatformDepBench.unsafeBytesEqual       4  avgt   10     4.263 ±   0.110  ns/op
PlatformDepBench.unsafeBytesEqual      10  avgt   10     5.206 ±   0.133  ns/op
PlatformDepBench.unsafeBytesEqual      50  avgt   10     8.160 ±   0.320  ns/op
PlatformDepBench.unsafeBytesEqual     100  avgt   10    13.810 ±   0.751  ns/op
PlatformDepBench.unsafeBytesEqual    1000  avgt   10    89.077 ±   7.275  ns/op
PlatformDepBench.unsafeBytesEqual   10000  avgt   10   773.940 ±  24.579  ns/op
PlatformDepBench.unsafeBytesEqual  100000  avgt   10  7546.807 ± 110.395  ns/op

After

Benchmark                          (size)  Mode  Cnt     Score     Error  Units
PlatformDepBench.unsafeBytesEqual       4  avgt   10     3.337 ±   0.087  ns/op
PlatformDepBench.unsafeBytesEqual      10  avgt   10     4.286 ±   0.194  ns/op
PlatformDepBench.unsafeBytesEqual      50  avgt   10     7.817 ±   0.123  ns/op
PlatformDepBench.unsafeBytesEqual     100  avgt   10    11.260 ±   0.412  ns/op
PlatformDepBench.unsafeBytesEqual    1000  avgt   10    84.255 ±   2.596  ns/op
PlatformDepBench.unsafeBytesEqual   10000  avgt   10   591.892 ±   5.136  ns/op
PlatformDepBench.unsafeBytesEqual  100000  avgt   10  6978.859 ± 285.043  ns/op
[maven-release-plugin] copy for tag netty-4.1.34.Final
Jenkins job that publishes netty artifacts is splitted in two subjobs.
First publishes OSX artifacts, seconds publishes all other artifacts.
The former depends on the latter but this dependency is not reflected
in jenkins. During a first run OSX subjob fails due to unresolved
dependencies (artifacts published by the second subjob).
Add -U to workaround caching "unresolved dependencies".
Motivation:

The special case fixed in netty#8497 also requires that we keep a derived slice when trimming components in place, as done by the capacity(int) and discardReadBytes() methods.

Modifications:

Ensure that we keep a ref to trimmed components' original retained slice in capacity(int) and discardReadBytes() methods, so that it is released properly when the they are later freed. Add unit test which fails prior to the fix.

Result:

Edge case leak is eliminated.
Copy link
Author

@sbtourist sbtourist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent job, just left a few minor comments to address.

for (;;) {
long usedMemory = DIRECT_MEMORY_COUNTER.get();
long newUsedMemory = usedMemory + capacity;
if (DIRECT_MEMORY_COUNTER != null) {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've made sure DIRECT_MEMORY_COUNTER is always non-null, so this check is confusing.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do

@@ -658,15 +673,13 @@ public static void freeDirectNoCleaner(ByteBuffer buffer) {
}

private static void incrementMemoryCounter(int capacity) {
for (;;) {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm on the fence about this change: moving from a CAS loop to the double addAndGet() might be faster, but by doing so, a single big-ish allocation which goes over the limit could cause concurrent small allocations to fail as well before the memory limit is brought back below threshold via DIRECT_MEMORY_COUNTER.addAndGet(-capacity).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I'll revert.

@@ -78,7 +78,7 @@ protected void cancelScheduledTasks() {
}

final ScheduledFutureTask<?>[] scheduledTasks =
scheduledTaskQueue.toArray(new ScheduledFutureTask<?>[scheduledTaskQueue.size()]);
scheduledTaskQueue.toArray(new ScheduledFutureTask<?>[0]);
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While you're at this, could you make scheduledTaskQueue protected? See DB-3884.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do. Just out of curiosity, how will this help in DB-3884?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. No final plans yet, but we might need to peek at the scheduled queue ourselves.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review remark fixes are pushed to a separate PR #19

jtgrabowski and others added 8 commits December 10, 2019 15:38
- Reason for reverting incrementMemoryCounter implementation (from
Sergio's comment):

"..moving from a CAS loop to the double addAndGet() might be faster, but
by doing so, a single big-ish allocation which goes over the limit
could cause concurrent small allocations to fail as well before the
memory limit is brought back below threshold via
DIRECT_MEMORY_COUNTER.addAndGet(-capacity)"

- scheduledTaskQueue is now protected to simplify DB-3884

"we might need to peek at the scheduled queue ourselves"
…netty#9492)

Motivation:

`HttpObjectDecoder` pre-checks that it doesn't request characters
outside of the `AppendableCharSequence`'s length. `0` is always allowed
because the minimal length of `AppendableCharSequence` is `1`. We can
legally skip index check by using
`AppendableCharSequence.charAtUnsafe(int)` in all existing cases in
`HttpObjectDecoder`.

Modifications:

- Use `AppendableCharSequence.charAtUnsafe(int)` instead of
`AppendableCharSequence.charAt(int)` in `HttpObjectDecoder`.

Result:

No unnecessary index checks in `HttpObjectDecoder`.

(cherry picked from commit 85fcf4e)
…30#section-3.2.4 (netty#9585)

Motivation:

When parsing HTTP headers special care needs to be taken when a whitespace is detected in the header name.

Modifications:

- Ignore whitespace when decoding response (just like before)
- Throw exception when whitespace is detected during parsing
- Add unit tests

Result:

Fixes netty#9571

(cherry picked from commit 39cafcb)
)

Motivation:

Technical speaking its valid to have http headers with no values so we should support it. That said we need to detect if these are "generated" because of an "invalid" fold.

Modifications:

- Detect if a colon is missing when parsing headers.
- Add unit test

Result:

Fixes netty#9866

(cherry picked from commit a7c18d4)
…-length and transfer-encoding: chunked header when using HTTP/1.1 (netty#9865)

Motivation:

RFC7230 states that we should not accept multiple content-length headers and also should not accept a content-length header in combination with transfer-encoding: chunked

Modifications:

- Check for multiple content-length headers and if found mark message as invalid
- Check if we found a content-length header and also a transfer-encoding: chunked and if so mark the message as invalid
- Add unit test

Result:

Fixes netty#9861

(cherry picked from commit 8494b04)
Compared against 4.1.34.2.dse, this tag cherry-picks upstream commits
that fixed bugs in HttpObjectDecoder/HttpRequestDecoder, plus two
intermediate refactoring commits that indirectly affect those bugfix
commits.

What follows is a list of PR links, issue links, CVE links, and hashes
associated with the cherry-picked commits.

Verify we do not receive multiple content-length headers or a content-length and transfer-encoding: chunked header when using HTTP/1.1 (netty#9865)
	https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-7238
	netty#9861
	netty#9865

	8494b04

Detect missing colon when parsing http headers with no value (netty#9871)
	https://nvd.nist.gov/vuln/detail/CVE-2019-20444
	netty#9866
	netty#9871

	a7c18d4

Fix typos in javadocs (netty#9527)
	skipped

Correctly handle whitespaces in HTTP header names as defined by RFC7230#section-3.2.4 (netty#9585)
	https://nvd.nist.gov/vuln/detail/CVE-2019-16869
	netty#9571
	netty#9585

	39cafcb

Use `AppendableCharSequence.charAtUnsafe(int)` in `HttpObjectDecoder` (netty#9492)
	netty#9492

	85fcf4e
DB-4068 cherry-pick upstream HttpRequest/ObjectDecoder fixes (4.1.34/bdp master)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.